Since the single-criterion-based tracker can not adapt to the complex environment, a tracking approach based on adaptive fusion of dual-criteria was proposed. In the method, the second-order spatiogram was employed to represent the target, the similarity between the target candidate and the target model as well as the contrast between the target candidate and its neighboring background were used to evaluate its reliability, and the objective function (or likelihood function) was established by weighted fusion of the two criteria. The particle filter procedure was used to search the target, and the fuzzy logic was applied to adaptively adjust the weights of the similarity and contrast. Experiments were carried out on several challenging sequences such as person, animal, and the results show that, compared with other trackers such as incremental visual tracker, ι₁ tracker, the proposed algorithm obtains better comprehensive performance in handling occlusion, deformation, rotation, and appearance change, and its success rate and average overlap ratio are respectively more than 80% and 0.76.